Filter bank based phase correlation architecture for motion estimation

ABSTRACT

Systems and methods for identifying motion between a previous frame and a current frame. The system may include a fast Fourier transform calculator that generates low pass frequency domain outputs and high pass frequency domain outputs of previous frame data and current frame data. The system may further include a phase difference calculator that calculates a first phase difference between the low pass frequency domain outputs and a second phase difference between the high pass frequency domain outputs. An inverse Fourier transform calculator may be included to generate a first inverse Fourier result and a second inverse Fourier result based on the first and second phase difference respectively, and a motion vector calculator may be included for generating motion vectors based on the inverse Fourier results.

This application claims the benefit of U.S. Provisional application Ser.No. 61/051,782, filed 9 May 2008; U.S. Provisional Application No.61/046,922, filed 22 Apr. 2008; and U.S. Provisional Application No.61/059,524, filed 6 Jun. 2008, all of which are herein incorporated intheir entirety by reference. This application is related to U.S.Non-provisional application Ser. No. 12/400,227, entitled “Picture RateConversion System Architecture” filed on the same day as thisapplication and U.S. Non-provisional application Ser. No. 12/400,220,entitled “Picture Rate Conversion System for High Definition Video,”also filed on the same day as this application.

FIELD

The technology described in this patent document relates generally tothe field of picture rate conversion, and more particularly to motioncompensated picture rate conversion.

BACKGROUND

Typical movie films are recorded at 24 Hz, 25 Hz or 30 Hz. Picture ratesof common video cameras are 50 Hz and 60 Hz. Commercially availabletelevision displays, on the other hand, have picture rates up to andbeyond 120 Hz, and employ either progressive or interlaced scanning.Hence, to interface broadcast video with a high-end TV display, theoriginal sequence from the broadcast video needs to be up-convertedusing, for example, a picture rate converter. A picture rate convertertypically operates by interpolating image frames at time instances wherethe frame sequence from a lower-frequency source device has yet to beregistered in a higher-frequency destination display.

In simple picture rate converters, a picture is often repeated in thedestination display until the next picture arrives from the sourcedevice, which oftentimes results in blur and judder when motion occurs.Motion estimation and compensation circuits may be used in a picturerate converter to reduce these unwanted effects and achieve a highperformance conversion for moving sequences. Motion compensationoperates by estimating where elements of an interpolated picture wouldbe, based on the direction and speed of the movement of those elements.The direction and speed values may then be expressed as motion vectorsand are used to “move” the elements to the correct position in the newlyinterpolated frame. If this technique is applied correctly, its impactmay be immediately visible on any picture sequence involving motion,where the resulting pictures can hardly be distinguished from theoriginal sequences before the up-conversion.

It is thus desirable to provide methods and systems that minimizecomputational cost associated with motion-compensated picture rateconversion while maximizing its estimation accuracy.

SUMMARY OF THE INVENTION

In accordance with the teachings provided herein, systems and methodsare provided for identifying motion between a previous frame and acurrent frame. A fast Fourier transform calculator may be included togenerate low pass frequency domain outputs and high pass frequencydomain outputs from the previous frame data and current frame data. Aphase difference calculator may calculate a first phase differencebetween the low pass frequency domain outputs and a second phasedifference between the high pass frequency domain outputs.

An inverse Fourier transform calculator may generate a first inverseFourier result and a second inverse Fourier result based on the firstand second phase difference respectively, and a motion vector calculatormay generate motion vectors based on the inverse Fourier results.

A system may also include a decomposition filter that receives previousframe data and current frame data where the decomposition filterincludes a high pass filter and a low pass filter. The system mayoperate on previous frame data and current frame data that correspond toframe-blocks 64×32 in size where the low pass filter is a multi-tapfilter having a block size of 15×7. The high pass filter may be agradient edge based detector, such as a Sobel edge detector, and thefast Fourier transform calculator may include at least two fast Fouriertransform calculation modules, which may utilize a radix 2decimation-in-time implementation. The fast Fourier transform calculatormay implement an input resolution of 2×16 bits and have an outputresolution of 2×16 bits where storage for the low pass frequency domainoutputs has allocations for a maximum of half of the columns based onadvantages of conjugate symmetry.

The system may include a phase difference calculator that uses a CORDICimplementation where the first phase difference and the second phasedifference are represented with 8 bit resolution, and the phasedifference calculator has a latency of 6 clock cycles. The phasedifference calculator may include a phase subtractor and a lookup table,and the inverse Fourier transform calculator may utilize hardware thatis also used by the fast Fourier transform calculator.

In the above system, the first inverse Fourier result may be a firstphase plane correlation surface, and the second inverse Fourier resultmay be a second phase plane correlation surface. The motion vectorcalculator may include a peak searcher that identifies a first peak onthe first phase plane correlation surface and a second peak on thesecond phase plane correlation surface. Two motion vectors may begenerated corresponding to a location of the first identified peak and alocation of the second identified peak where the location of the firstidentified peak and the second identified peak correspond to a movementin a video from the previous frame to the current frame. The peaksearcher may be configured to identify a first plurality of peaks on thefirst phase plane correlation surface and a second plurality of peaks onthe second phase plane correlation surface.

As a further example, a method of identifying motion between a previousframe and a current frame may include decomposing received previousframe data and current frame data into a high pass component and a lowpass component and performing a fast Fourier transform of the high passcomponents and low pass components. A first phase difference between thehigh pass component of the current frame data and the high passcomponent of the previous frame data may be calculated, and a secondphase difference between the low pass component of the current framedata and the low pass component of the previous frame data may also becalculated. An inverse fast Fourier transform may be performed on thefirst phase difference and the second phase difference, and a firstmotion vector and a second motion vector may be calculated based on thefirst inverse Fourier result and the second inverse Fourier result.

As a further example, a method of identifying motion between a previousframe and a current frame may include reading a block of current framelow pass data and performing a first fast Fourier transform of thecurrent frame low pass data. The method may also include reading a blockof previous frame low pass data and performing a second fast Fouriertransform of the previous frame low pass data. A first phase differencemay be calculated between the results of the first fast Fouriertransform and the second fast Fourier transform, and a first inversefast Fourier transform may be performed of the calculated first phasedifference. The method may also include searching for and identifying afirst top value of an output of the first inverse fast Fourier transformand generating a motion vector from the identified first top value.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a motion compensated picture rateconverter (MCPRC) system configuration.

FIG. 2 is a block diagram illustrating another MCPRC systemconfiguration.

FIG. 3 is a block diagram illustrating a motion compensated picture rateconverter including a phase plane correlation calculator.

FIGS. 4A and 4B depict a block diagram illustrating components of anexample MCPRC system.

FIG. 5 is a flow diagram illustrating a phase plane correlation processfor generating motion vectors.

FIG. 6 is a block diagram illustrating a phase plane correlationcalculator configuration.

FIG. 7 is a timing chart illustrating buffer usage and timing of a phaseplane correlation calculation operation.

FIG. 8 is a block diagram illustrating another phase plane correlationcalculator configuration.

FIG. 9 is a block diagram illustrating inputs to a vector motionvalidation engine.

FIG. 10 is an illustration of a block structure for the reference andinterpolated frames.

FIG. 11 is a block diagram illustrating components of an example motionvector validation engine.

FIG. 12 is a flow diagram illustrating a process for calculating motionvectors from previous frame data and current frame data inputs.

FIG. 13 is a flow diagram illustrating a process for generating anintermediate frame from previous frame data, current frame data, andadditional motion vector inputs.

FIG. 14 is a block diagram illustrating a motion compensated picturerate converter including a phase plane correlation calculator and aglobal motion calculator.

FIG. 15 is a flow diagram illustrating the calculation of affineparameters in a global motion calculator.

FIG. 16 is a flow diagram illustrating a process for calculatingrefinement affine parameters.

FIG. 17 is a flow chart illustrating a process for calculatingrefinement affine parameters.

FIG. 18 is a flow diagram illustrating a process for generating anintermediate frame from previous frame data and current frame data.

FIG. 19 is an illustration of a motion hole.

FIG. 20 is a flow diagram illustrating an error concealment technique.

FIG. 21 is a further illustration of a motion hole requiring errorcompensation.

FIG. 22 is an illustration of texture synthesis during an errorconcealment operation.

FIG. 23 is an illustration of texture synthesis based on the Markovmodel.

FIG. 24 is a flow diagram illustrating a final image synthesis decision.

FIG. 25 is a flow diagram illustrating a process for generating aninterpolated frame from previous frame data and current frame data inputand deciding whether to use the interpolated frame in a target video.

DETAILED DESCRIPTION

FIG. 1 is a block diagram illustrating a motion compensated picture rateconverter (MCPRC) system configuration 10. An input signal, having adiscrete sequence of video frames, is input to the MCPRC system 10,which produces an up-converted, motion compensated output signal viamodules 12, 14, 16, and 18 of the MCPRC system 10. Each of the modulesof MCPRC system 10 is described herein. Subsequent to the up-conversion,the output signal from the MCPRC system 10 has a frame rate that istypically higher than the frame rate of the input signal. For example,the input signal may be produced from a video camera which has a picturerate of 60 Hz. This video signal may need to be up-converted using theMCPRC system 10 in order to be suitable for output on an LCD paneldisplay having, for example, a refresh rate of 120 Hz. In general, framerate up-conversion is achieved by injecting a pre-determined number offrames between every pair of temporally adjacent input frames. Theseintermediate frames may be created to approximately capture motiontrajectories of objects between frames, thereby enhancing the overallsmoothness of a video image sequence as it is displayed afterup-conversion.

With reference to FIG. 1, the input signal is first processed by a frontend module 12. The front end module 12 may contain components such astuners, demodulators, converters, codecs, analog video decoders, etc. Anoutput from the front end module 12 is then passed downstream to a videoformat conversion and processing module 14 which may apply additionalvideo enhancements such as reducing analog noise and compressionartifacts, scaling, OSD blending, color management, etc. The resultingoutput is subsequently fed to a motion compensated picture rateconversion (MCPRC) module 16 producing an output that is processed by atiming controller 18 before being propagated as an output signal.

The entire MCPRC system 10 illustrated in FIG. 1 may be implemented on asingle chip. In one example, an MCPRC chip may be incorporated intotelevision circuitry where an up-converted, post-processed output of theMCPRC system is transmitted to an external display panel for video.However, the MCPRC architecture 10 of FIG. 1 may also be implemented ona plurality of chips wherein the front end 12 and video formatconversion and processing module 14 are fabricated on one chip and amotion compensated picture rate conversion module 16 and timingcontroller 18 are fabricated on a separate chip. Additional chipcombinations of the elements of FIG. 1 may also be implemented.

FIG. 2 is a block diagram illustrating another MCPRC systemconfiguration 30. In the configuration of FIG. 2, some of the processingof the video format conversion and processing module 14 of FIG. 1 hasbeen moved to a post processing module 36. In this arrangement, an inputsignal is received by a front end module 32. The front end module 32 maycontain components such as tuners, demodulators, converters, codecs,analog video decoders, etc. An output from the front end module 32 ispassed downstream to a noise reduction and de-interlacing module 34which may, for example, convert the input signal from a native interlacescan-based form to a high quality progressive scan output whilerealizing a significant reduction in analog noise and compressionartifacts such as block noise and mosquito noise. The resulting outputmay be subsequently fed to a motion compensated picture rate conversionmodule 16, which generates motion compensated interpolated frames toproduce a video output sequence. The MCPRC module 16 will be describedbelow in further operational detail. The video output is then processedby a post-processing module 36 that may apply additional videoenhancement functions to the video signal, such as scaling, edgeenhancement, color management, picture controls, etc.

Like the configuration of FIG. 1, the example of FIG. 2 may beimplemented on a single chip or a plurality of chips. The postprocessing module 36 may be included on the same chip as the MCPRC 16 ormay be used as a separate chip in the FIG. 2 architecture, for example.In certain configurations, modules 32, 34, 16, and 36 are structurallysimilar to their counterpart modules 12, 14, 16 and 18 of FIG. 1.

As illustrated in FIGS. 1 and 2, video information channels 20 and 38,respectively, are provided to facilitate the transfer of informationbetween the modules in the corresponding MCPRC systems 10 and 30. Inparticular, information that may be conveyed to the MCPRC modules 16includes, for example, the position of a closed caption display, thepresence of an on-screen display, the native frame rate of therespective input signals, and the origin and active video boundaries ofrespective input signals.

In the illustrated MCPRC systems 10 and 30, the input video signals mayrange from Standard Definition (NTSC/PAL/SECAM) to High Definition andmay be interlaced or progressive-based. In some instances, the videosignal resolution is even lower than Standard Definition with low framerates. For example, the input video signal may be a QVGA (320×240) inputat 15 or 30 frames per second from a connector device in a portablemedia player such as an iPod. In certain instances, the low-resolutionvideo signal may be fed to a video dock in a personal media player or amultimedia cellular phone via a connector device, where the dock maycontain an integrated circuit capable of performing spatial and temporalconversions from, for example, 320×160 at 5 fps to 720×480 at 60 fps.Interlaced inputs may be composed of video-originated or film-originatedmaterial. Video-originated material may be first de-interlaced andconverted from a field rate to a frame rate before being input to MCPRCmodules 16.

FIG. 3 is a block diagram illustrating a motion compensated picture rateconversion module including a phase plane correlation calculator thatmay be integrated as module 16 in the configurations of FIGS. 1 and 2,respectively, for providing object motion estimation between pairs ofconsecutive frames in an input video signal 48. For each pair ofconsecutive frames to be interpolated, the earlier of the two frames isreferred to as a “previous frame,” and the latter of the frames isreferred to as a “current frame.”

In FIG. 3, the input video signal 48 is processed by a phase planecorrelation calculator 42. The phase plane correlation module 42determines a motion vector estimate 52 that is provided to an SAD basedvector validation module 44. The SAD based vector validation module 44also may receive a copy of the input video signal 48 and a videoinformation channel 50. The SAD based vector validation module 44determines a per pixel motion vector 54, 58 based upon the motion vectorestimate 52 from the phase plane correlation calculator 42 and mayconsider other motion vector inputs (not shown) in that determination.The per pixel motion vectors are received by an interpolation and errorconcealment module 46 where intermediate frames are generated forinsertion into a target video.

FIGS. 4A and 4B depict a block diagram illustrating components of anexample MCPRC system 60. In the MCPRC system 60 of FIG. 4, a two-channelLVDS input receiver 62 receives a two-channel LVDS input signal andprovides the input signal to a formatter 68 that executes deserializingoperations. The formatter 68 passes a deserialized output to physicalmemory 72 via a DDR SDRAM controller 70. The formatter 68 may alsoprovide the deserialized output to an RGB to luminance converter 74. Apair of luminance filters 76 receive the converted signal from the RGBto luminance converter 74. The luminance filters 76 decompose thereceived signal into a low pass representation and a high passrepresentation of the received input signal. The low pass representation(YLPF_PPC) and the high pass representation (YHPF_PPC) are provided tothe physical memory 72 via the DDR SDRAM controller 70 for storage.

Following storage of the low pass and high pass representations of theinput signal, a phase plane correlation calculator (PPC) 78 accesses thefiltered representations of the input signals and performs a phase planecorrelation calculation on the filtered data. In performing the phaseplane correlation calculation, the PPC 78 may utilize a buffer 80 and aprocessing unit 82 such as an Xtensa™ CPU. The PPC 78 and processingunit 82 perform a phase plane correlation calculation on the filteredrepresentations of the input signal to produce one or more candidatemotion vectors. The candidate motion vectors may be stored in a validatemotion vector (VMV) buffer 84 prior to access by a VMV engine 86.

The VMV engine 86 receives candidate motion vectors from the VMV buffer84 as well as previous frame data and current frame data from physicalmemory 72 via the DDR SDRAM controller 70. The VMV engine 86 determinesa final motion vector to be used in generating one or more intermediateframes by comparing the candidate motion vectors stored in the VMVbuffer 84 to the received previous frame data and current frame data.For example, the VMV engine 86 may perform a quality calculation foreach of the candidate motion vectors and select the candidate motionvector having a best score relating to the accuracy of the candidatemotion vector in depicting object motion from the previous frame to thecurrent frame while offering a high degree of picture smoothness.Following selection of a final motion vector based on the qualitycalculation, the VMV engine 86 generates one or more intermediate framesbased on the selected final motion vector for insertion into the finalup-converted target video.

In calculating motion vectors for integrating intermediate frames forup-conversion of video, some portions of the intermediate frame may beundefined by the final motion vectors selected by the VMV engine 86.This anomaly in motion based interpolation often occurs at edges ofobjects appearing in the video. This phenomenon is illustrated in FIG.19, which will be discussed in further detail below. To account forthese undefined regions, the MCPRC system 60 may include a compensationand dynamic averager 88. The compensation and dynamic averager 88receives intermediate frames generated by the VMV engine 86 and performserror correction techniques on the intermediate frames. These errorcorrection techniques fill in the portions of the intermediate framesleft undefined by the frame generation procedures as will be describedin detail below. The compensation and dynamic averager 88 processes theintermediate frames and outputs the error corrected intermediate framesto physical memory 72.

Following generation of the intermediate frames and error correction, asecond formatter 90 reads the target video, which includes previousframe data, current frame data, and the generated intermediate frames,and converts them into the desired output format. In the example of FIG.4, the formatted output is then propagated to LVDS buffers 92 whichoutput the target video in a quad-channel LVDS configuration 94 as shownat 96, 98, 100, and 102.

It should be noted that various modifications may be made to the exampleof FIG. 4. These modifications could include the omission of one or moreof the depicted elements, rearrangement of the depicted components, aswell as the inclusion of additional circuitry such as synchronizationelements 110, buffer pointer controls 112, phase lock loops 108, clockgenerators 108, register configuration modules 104, CPU code ROM 106,etc. Storage in physical memory 72 of inputs and outputs may or may notbe done by each of the depicted circuit elements or may not be done atall. Additionally, some or all portions of the calculations may beperformed on block units that make up a portion of a video frame at atime due to the high pixel density of modern video. For example, asystem performing picture rate conversion for a video having 1920×1080pixels may operate on individual blocks 64×32 pixels in size and thencoordinate results of the block calculations in generating intermediateframes.

FIG. 5 is a flow diagram illustrating a phase plane correlationcalculator 120 for generating motion vectors. The phase planecorrelation calculator 120 calculates the motion between two framesusing the Fourier shift theorem. The Fourier shift theorem states thattwo signals shifted by a uniform translation are phase shifted whenrepresented in the frequency-domain. Thus, by taking the Fouriertransform of two images that are shifted by a given translation, theFourier representation of the signal is shifted by a phase shift that isproportional to the translation. The inverse Fourier transform of thephase shift generates a phase correlation surface where the position ofthe peaks in the surface represents the magnitude of the shift, and theheight of the peaks represents the reliability of the estimated motionvectors. The maximum translation that a phase plane correlationcalculator 120 is capable of detecting is based on the size of theFourier transform. An N point Horizontal by M point vertical Fouriertransform can measure the maximum shift of +/− N/2 horizontal and +/−M/2 vertical. The typical motion range for a 1080p resolution signal mayuse a Fourier transform of 64×32 pixels or more.

When block sizes becomes large, the reliability of the estimated motionvectors may decrease. Thus, it is often possible to miss small objectmotion because small objects do not make large contributions to thecorrelation surface and are masked by noise in the image. To circumventthis problem, a filter bank based design may be utilized. Filtering theinput signal into low pass representations and high pass representationsaides the MCPRC system in identifying both large and small object motionwithin a video frame. Typical motion compensation converters are unableto account for such multiple object movement within a video frame. Thus,incorrect motion vectors may be calculated where multiple objects aremoving at the same time, such as in a sports video where players mayappear as large objects on the screen moving in one direction while asmall ball may also be depicted moving in a different direction or at adifferent speed. By decomposing input signals into both low pass andhigh pass representations, both small and large object motion may bebetter accounted for and more optimal motion compensation may beaccomplished. The low pass filtered image captures the global motion orlarge object motion in the block, and the high pass filtered imagecaptures the small object motion. Because these two motion engines areindependent of each other, the problem of failing to compensate forsmall object motion may be addressed.

The process for generating motion vectors of FIG. 5 begins with thereceipt of previous frame data and current frame data for a block at afilter band based decomposition and quantization unit 122. In the filterband based decomposition and quantization unit 122, the effect of noiseis minimized by quantization of the signal, and the previous frame dataand current frame data are decomposed into high pass 124 and low pass126 representations via a high pass filter and a low pass filterresulting in a low pass previous frame representation (F₁(x₁,y₁)), a lowpass current frame representation (F₂(x₂,y₂)), a high pass previousframe representation (F₃(x₃,y₃)), and a high pass current framerepresentation (F₄(x₄,y₄)).

Following decomposition, each of the representations are processed byone or more two-dimensional fast Fourier transform calculators (FFTs)128, 130. The two-dimensional FFTs 128, 130 take the time-domainrepresentations output by the filter band based decomposition andquantization unit 122 and convert the representations intofrequency-domain representations: F₁(ω_(x), ω_(y)), F₂(ω_(x), ω_(y)),F₃(ω_(x), ω_(y)), F₄(ω_(x), ω_(y)). Some or all of the frequency-domainrepresentations may be temporarily stored in a frame buffer 136 beforeproceeding with further calculation. Maximum values at the beginning ofa block may be retained and stored for later detection of periodicstructures as shown at 132.

Following calculation of the frequency-domain conversions, a phasedifference 138 is calculated between the low pass, frequency-domainrepresentations of the previous frame data and the current frame data.For example, the phase difference may be calculated by solving for the“A” and “B” parameters of the following formula:F ₂(ω_(x),ω_(y))=e ^(−j(Aω) ^(x) ^(+Bω) ^(y) ⁾ F ₁(ω_(x),ω_(y)).After calculating the phase difference 138 between the previous framedata and the current frame data, a two-dimensional inverse fast Fouriertransform (IFFT) 140 is applied to the calculated phase difference 138.The result of the IFFT 140 calculation is a two-dimensional phase planecorrelation surface. The phase plane correlation surface may be viewedas a contour map identifying motion between the previous frame and thecurrent frame of the source video. The locations of peaks on the phaseplane correlation surface (a₁, b₁) correspond to motion within the frameblock such that:F ₂(x ₂ ,y ₂)=F ₁(x ₂ +n·a ₁ ,y ₂ +m·b ₁).The height of a peak on the phase correlation surface corresponds to thesize of an object that is moving within a block. To locate peaks withinthe phase correlation surface, a peak search 142 is performed, and basedon the identified peaks, a low pass filter based motion vector 144 isdetermined. The low pass filter based motion vector 144 corresponds tolarge object motion within a frame block.

A similar process is performed utilizing the high pass frequency-domainrepresentations (F₃(ω_(x), ω_(y)), F₄(ω_(x), ω_(y))) calculated by thetwo-dimensional FFT 128. Again, some or all of the high passfrequency-domain representations may be temporarily stored in a framebuffer 136. Following calculation of the high pass frequency-domainrepresentations, a phase difference 146 is calculated between the highpass, frequency-domain representations of the previous frame data andthe current frame data. For example, the phase difference 146 may becalculated by solving for the “C” and “D” parameters of the followingformula:F ₄(ω_(x),ω_(y))=e ^(−j(Aω) ^(x) ^(+Bω) ^(y) ⁾ F ₃(ω_(x),ω_(y)).After calculating the phase difference 146 between the previous framedata and the current frame data, a two-dimensional IFFT 148 is appliedto the calculated phase difference 146. The result of the IFFTcalculation 148 may be viewed as a second two-dimensional phase planecorrelation surface. The locations of peaks on the second phase planecorrelation surface (c₁, d₁) correspond to motion within the frame blocksuch that:F ₄(x ₄ ,y ₄)=F ₃(x ₄ +n·c ₁ ,y ₄ +m·d ₁).To locate peaks within the second phase correlation surface, a peaksearch 150 is performed, and based on the identified peaks, a high passfilter based motion vector 152 is determined. The high pass filter basedmotion vector 152 corresponds to small object motion within a frameblock.

One of the difficulties in performing motion compensation on videohaving a high pixel density is the number of computations to beperformed to accomplish high quality compensation. A video having1920×1080 pixels is made up of about 1000 64×32 pixel blocks where eachpixel block includes over 2000 pixels. Thus, an input picture rate of 60Hz results in about 120 million calculations per second of video. Toaccommodate this large number of calculations, care may need to be takenregarding selection of components for the phase plane correlationcalculator 42.

The choice of the filter bank involves appropriate selection of the lowpass and high pass filter. It may be desirable to select filtersaccording to the input signal. For example, for a block size of 64×32pixels a multi-tap filter of 15×7 pixels may be selected. The filtersize may also be selected to be proportional to the input resolution ofthe video. A higher resolution video has more redundancy of data and mayrequire a larger filter kernel. Gradient edge based detectors such asthe Sobel edge detector have been discovered to perform well as highpass filters. The gradient edge based detectors reliably detect thepresence of small objects in blocks. To facilitate simplified,high-speed calculations by the FFTs 128, 130, the low pass and high passdata may be quantized to two bit and one bit resolution respectively.

As stated above, the FFTs 128, 130 may be required to perform very largenumbers of computations very quickly. Examples of FFTs that may be usedin the phase plane correlation calculator 42 include an FFT module thatcomputes a 64 point FFT using a radix 2 decimation-in-time (DIT)implementation. The internal resolution of the FFT and hence the inputand output resolutions are 2×16 bit wide (real and complex data). TheFFT module is designed in a pipelined fashion and has a latency ofaround 10 clock cycles. Thus, if the input 64 data points are ready inclock 1, then the output 64 FFT values will be ready after 10 clockcycles latency. This design has a very high throughput. The inverse FFTmay be done using the same block as is used for FFTs using a signal thatcontrols the sign specific multipliers in the butterfly structure of theFFT module. A 32 point FFT may also be done using this FFT module byalternate zero insertion of the input data and then taking only thefirst 32 points of the output. The minimum time taken to compute thehorizontal FFTs (HFFT) in this design of a 64×32 block is 32+10 clockcycles and for vertical FFTs (VFFT) is 64+10 clocks. The phasedifference calculators 138, 146 may be implemented by modules using aCORDIC implementation to calculate the phase and magnitude of a complexnumber.

Following an IFFT calculation, the phase plane correlation surface haspeaks that correspond to motion in the block. The phase planecorrelation calculation process may result in multiple peaks in thecorrelation surface. An example search process 142, 150 finds the topfour values of peaks in the correlation surface. Due to the presence ofsub-pixel motion it is possible to have peaks whose values are sharedbetween two adjacent locations. Thus, the search process may calculate aunique peak value in a neighborhood of 8 positions. In other words if apeak is found in position (x,y), then nearby peaks in positions(x+/−1,y+/−1) may not be considered. The location and magnitude of thediscovered peaks are then utilized to calculate low pass and high passbased motion vectors 144, 152.

FIG. 6 is a block diagram illustrating components of an example phaseplane correlation calculator 160 for generating motion vectors. Theimplementation of FIG. 6 includes five buffers 164, 166, 168, 170, 172,a controller 162, and three data processing blocks 174, 176, and 178.Buffer A 164 and buffer B 166 are responsive physical memory. The inputdata to the phase plane correlation calculator 160 is the low pass andhigh pass filtered data from the current and previous frames which arereferred to in the descriptions of FIGS. 6-8 as CLP (current frame lowpass data), CHP (current frame high pass data), PLP (previous frame lowpass data), and PHP (previous frame high pass data). The FFT module 174may be multiplexed to perform both HFFT and VFFT operations as well asinverse FFT operations. The controller 162 is responsible for timing thedata flow inside the phase plane correlation calculator 160. Theconfiguration 160 of FIG. 6 is able to take advantage of the conjugatesymmetry of FFTs of real input data resulting in a potential memorysavings if downstream calculation compensations are made.

FIG. 7 is a timing chart depicting the timing of calculations and bufferusage during an example operation of the phase plane correlationcalculator of FIG. 6. In the example of FIG. 7, the vertical axisdisplays time in units of clock cycles, and the horizontal axisrepresents individual buffers that correspond to buffer A 164, buffer B166, buffer N 168, buffer O 170, and buffer P 172 of FIG. 6. The FFTUsage column 182 identifies what type of operation the FFT module 174 isperforming. Similarly, the Phase Extraction column 184 identifies anyphase extraction operations 176 being performed at a particular time. Inthe example of FIG. 7, a latency of 100 clock cycles is associated withreading a 64×32 block of data from physical memory. HFFTs take 50 clockcycles, VFFTs plus phase extractions take 50 clock cycles if 32 phaseextractors work in parallel, vertical inverse fast Fourier transforms(VIFFTs) plus phase difference calculations take 50 clock cycles if 32phase differences work in parallel, horizontal inverse fast Fouriertransforms (HIFFTs) take 50 clock cycles, data copies between bufferstake 50 clock cycles, and search operations can be accomplished in 150clock cycles utilizing 16 search engines operating in parallel whereeach row can be searched in 64 clock cycles (128 clock cycles pluslatency). In the example of FIG. 7, input data resolution is two bitsfor the low pass data and one bit for the high pass data, the output ofany FFT is 32 bits (16 bits to represent the real component, 16 bits torepresent the complex component), and phase values are 8 bits.

The phase plane correlation calculator 160 of FIG. 6 begins after 32lines of the current frame have been written into physical memory. Once32 lines of the current frame have been written into physical memory,the following steps are performed as depicted in FIG. 7. At clock cycle0, the block of CLP (64×32 samples) data is read from physical memoryand stored in buffer A 164. This step takes 100 clock cycles. At clockcycle 100, an HFFT of the rows of data in buffer A 164 is performed withthe results being simultaneously stored in buffer B 166. This is done in50 clock cycles. The data from buffer B 166 (a row wise databank) iscopied to buffer N 168 (a column wise databank) in 50 clocks at clockcycle 150 performing a transpose operation. This is to facilitate acolumn wise data read for the VFFT. At clock cycle 200, a VFFT isperformed on the data in buffer N 168 and a phase extraction 176 of the33 columns in buffer N 168 is performed with the phase value resultsbeing simultaneously stored in buffer P 172. This step takes 50 clockcycles.

Simultaneously at clock cycle 200, a block of PLP data can be read fromphysical memory to buffer A 164. In similar fashion as the CLP data, anHFFT of the PLP data is performed at clock cycle 300 with the resultsbeing stored in buffer B 166. A transpose operation is performed atclock cycle 350 transferring the PLP data from buffer B 166 to buffer N168. A VFFT is performed on the data in buffer N 168 at clock cycle 400with the results being stored in buffer O 170.

Following phase extraction for the PLP data, the phase differences ofthe data in buffer P 172 and buffer O 170 are calculated column wise andthen an inverse VFFT is performed with the results being stored inbuffer N 168 as illustrated at clock cycle 450. Data from buffer N 168is then copied to buffer B 166 at clock cycle 500 to facilitate row wiseprocessing. This takes 50 clock cycles. An HIFFT operation is performedon the data in buffer B 166 with the results stored in buffer A 164 atclock cycle 550. This takes 50 clock cycles. Due to the conjugatesymmetry memory savings procedures discussed above, only 33 samples ofthe row are available in buffer B 166. The remaining samples aregenerated by taking the complex conjugate of the existing columns andthe full 64 samples are sent for HIFFT. The 64×32 output samples of theHIFFT represent the phase plane correlation surface that are thensearched over the next 150 clock cycles beginning at clock cycle 600 tofind the top 4 values corresponding to potential peak values of thephase plane correlation surface. Candidate motion vectors are thendetermined based upon the identified peaks, and the above describedsteps are repeated for the high pass data.

FIG. 8 is a block diagram illustrating another example phase planecorrelation calculator configuration 190 for generating one or moremotion vectors. The configuration of FIG. 8 includes a multiplexermodule 196 that is responsive to two databanks A and B 200, a controller192, a search module 208, an FFT calculator 202, a phase extractionmodule 206, and a phase difference calculator 204. The phase planecorrelation calculator receives the CLP, CHP, PLP, and PHP data asinputs 198.

The controller 192 of FIG. 8 performs several functions. The controller192 generates read signals to physical memory that initiate datatransfer to the phase plane correlation calculator 190. The controller192 also generates selection logic for the multiplexer 196. Thecontroller 192 further generates the address and write enable signalsfor each memory, (e.g., sRAM) in databanks A and B 200. The controller192 may run according to a synchronous counter that is reset by a framereset signal 194. The phase plane correlation calculator 190 may run onthe system clock where all signals generated by the controller 192 aredependent on the state of the counter.

The inputs and outputs of the data processing modules 202, 204, 206 areconnected to the multiplexer module 196. Depending upon the operationbeing performed, individual multiplexers inside the multiplexer module196 are connected using selection logic from the controller 192. The FFTmodule 202 is multiplexed to perform HFFT and VFFT operations. Becausethe phase plane correlation calculator inputs are always real (noimaginary component), the columns of the block after performing an HFFToperation exhibit conjugate symmetry. Because this symmetry is knownbased on the real inputs, a memory savings can be realized because aftera VFFT operation, the columns from 64/2+1 onwards are circularly shiftedversions of columns 1 to 64/2. Due to this known data redundancy,storage for nearly half the columns may be eliminated as long ascompensations are made downstream based on this data paring. For exampledatabank A 200 may contain 33 sRAMs, 32 bits wide by 34 locations deep,sized to be able to handle the maximum storage and access requirements.Databank B 200 only stores phase values and can, therefore, be muchsmaller in size than databank A 200.

In the example of FIG. 8, after 32 lines of current frame data have beenwritten to physical memory, the phase plane correlation calculationprocessing can begin. The first 32 lines of data in physical memory areprocessed during the time taken for the next set of 32 lines to arrive.Thus, the data processing on a single 64×32 pixel block needs to befinished in 2048 clock cycles if one pixel is written to physical memoryper clock cycle. If the number of processing cycles is less than 2048,then the phase plane correlation calculator 190 may operate on a slowerclock.

In operation, the phase plane correlation calculator 190 of FIG. 8 readsa block of CLP (64×32 samples) data from physical memory, passes the CLPdata through the FFT module 202 for HFFT, and stores the first 33columns in databank A 200. The CLP HFFT data in databank A 200 istransposed to enable column reading. A VFFT is then performed on thedata in databank A 200 followed by a phase extraction 206 with theresulting phases being stored in databank B 200. While reading the datafrom the FFT, the FFT search 208 (for discovering periodic structures)is done for the CLP data.

PLP data is read from physical memory, passed through the FFT 202 forHFFT, and stored in databank A 200. The PLP HFFT data is transposed in Aas was done with the CLP HFFT data previously. A VFFT operation isperformed on the data in databank A 200 followed by phase extraction 206with the resulting phases being stored in databank A 200. The phasedifferences 204 of the data in databanks A and B 200 are calculatedcolumn wise, and the results of an inverse VFFT operation 202 on thephase differences are stored in databank A 200.

The data in databank A 200 is transposed to enable row processing, andan inverse HFFT is performed on the data in databank A 200 using the FFTmodule 202 with the results being stored in databank A 200. Whileperforming the HIFFT, only 33 samples of the row are available in B dueto the storage savings made possible based on the conjugate symmetry ofthe FFT of real input values as described above. The remaining samplesare generated by taking the complex conjugate of the existing columns,and the full 64 samples are sent to HIFFT. A PPC search is done on thedata in databank A 200 to find peak values in the phase planecorrelation surface, and corresponding output motion vectors aredetermined and output as illustrated at 210. These steps are thenrepeated for high pass data.

FIG. 9 is a block diagram 220 illustrating inputs to a vector motionvalidation (VMV) engine 222. The VMV engine 222 of FIG. 9 selects afinal motion vector describing object motion within a block from a setof candidate motion vectors, generates one or more intermediate frames,and performs error concealment techniques on the generated intermediateframes.

One method of selecting a final motion vector is by making a qualitycalculation for each of the candidate motion vectors and selecting thecandidate motion vector that yields the best score. Candidate motionvectors may come from estimation calculations such as the phase planecorrelation calculation described above, motion vectors from neighboringblocks, from other motion vector estimations such as the global motioncalculations described below, as well as a variety of other sources. Thequality calculation may include a variety of matching techniquesincluding sum of absolute differences (SAD) and sum of squareddifferences (SSD). The quality calculation may also include other termssuch as a smoothness term. For example, the following qualitycalculation formula may be utilized where the vector resulting in thelowest cost is selected as the final motion vector:Cost(x,y,t)=α∫f(Data)+β∫f(smoothness).The first function corresponds to a distortion measure such as SAD orSSD. If SAD is selected, the first function could be represented as:f(Data)=Σ|I ₁(x,y,t ₁)−I ₂(x−dx _(i) ,y−dy _(i) ,t ₂)|,where I₁ and I₂ are the current and previous frames, respectively, anddx_(i) and dy_(i) represent a candidate motion vector.

As noted above, picture rate up-conversion may be performed from avariety of source video rates to a variety of target data rates. Thefactor of the source to target data rates dictates the number ofintermediate frames that are interpolated into the target video. Forexample, an up-conversion from a 60 Hz source video to a 120 Hz targetvideo results in the insertion of 1 frame between frames of the sourcevideo in the target video. Thus, one frame is inserted halfway betweensource video frames resulting in an interpolation factor of 0.5: 60Hz/120 Hz=0.5. For conversion from a 24 Hz source video to 120 Hz targetvideo, four frames are inserted between source frames in the targetvideo. Inserting four frames between source video frames causes anintermediate frame to be inserted every 0.2 source frames resulting ininterpolation factors of 0.2, 0.4, 0.6, and 0.8: 24 Hz/120 Hz=0.2.

The interpolation factor is utilized in generating intermediate frames.A final motion vector selected by the VMV engine 222 corresponds to thedetected motion between the previous frame and the target frame.However, in the example of 60 Hz to 120 Hz conversion, the intermediateframe will depict object motion halfway between the previous frame andthe target frame. Thus, when calculating the proper motion of objectswithin a block in an intermediate frame, the final motion vector ismultiplied by the interpolation factor, 0.5, to capture object positionat the time of interest (i.e., the time of the intermediate frame).Similarly with 24 Hz to 120 Hz conversion, the first intermediate frameutilizes the final motion vector multiplied by the first interpolationfactor, 0.2, the second intermediate frame utilizes the final motionvector multiplied by the second interpolation factor, 0.4, the thirdintermediate frame utilizes the final motion vector multiplied by thethird interpolation factor, 0.6, and the fourth intermediate frameutilizes the final motion vector multiplied by the fourth interpolationfactor, 0.8.

FIG. 9 illustrates example inputs to the VMV engine 222. As describedabove, the VMV engine 222 receives one or more candidate motion vectors224. The VMV engine 222 may be configured to operate on differentsubsets of the entire picture data. For example, the VMV engine 222 mayoperate on single blocks at a time, a collection of blocks, or theentire picture. In the example of FIG. 9, the VMV engine 222 operates ona single slice, which in a 1920×1080 pixel environment having 64×32pixel blocks would be made up of a row of 30 64×32 pixel blocks. The VMVengine 222 receives the candidate motion vectors for a frame slice aswell as interpolation factors 226. The VMV engine 222 also receivesprevious frame data 230 in luminance form and current frame data 232 inboth luminance form 236 after a color space conversion 234 and in RGBform 238. The VMV engine 222 may also receive the previous frame data inRGB form (not shown). The VMV engine 222 outputs the RGB forminterpolated intermediate frames 240.

In operation, the example of FIG. 9 begins with an interrupt from thetiming controller to the CPU requesting candidate motion vectors 224 fora slice. The CPU responds by supplying the motion vectors 224 for theslice to the VMV engine 222. The number of motion vectors per block inthis example is 14. However, any number of motion vectors may beprovided to the VMV engine 222 provided sufficient computation time isavailable. The appropriate interpolation factor or factors 226 are alsoprovided to the VMV engine 222 at the direction of the CPU. The SAD iscalculated based on the vectors supplied to the VMV 222, the receivedprevious frame data 230 and current frame data 236, 238, and thetemporal and spatial smoothness constraint built into the cost function.The best motion vector is selected as the final motion vector, which isthen used to generate the interpolated frame.

FIG. 10 is an illustration of a block structure for the reference andinterpolated frames. The VMV engine 222 generates interpolated framesblock by block (e.g., one 64×32 pixel block at a time). To generate aninterpolated window, the VMV engine 222 utilizes pixel information fromboth the previous 230 and current 236, 238 frames. As shown in FIG. 10,the pixels in the generated window lie in spatially aligned locations inthe previous and current frames. Based on the maximum magnitude ofmotion vectors, the maximum motion supported may be limited. Forexample, the maximum motion supported may be ±32 pixels in the xdirection and ±16 pixels in the y direction. Thus, where “i” is theinterpolation factor, a pixel in the generated frame would be found at adistance of at most (32i, 16i) pixels in the previous frame and at adistance of at most (32(1−i), 16(1−i)) pixels in the current frame.

For 5× interpolation (e.g., 24 Hz to 120 Hz), as “i” increases from 0.2to 0.8, the boundary that defines the set of pixels used for SADcalculation increases in the previous frame and decreases in the currentframe. This fact may be used to optimize bandwidth for memory transfers.The VMV engine 222 may use the RGB representation from the frame thathas a smaller boundary and the luminance representation from the otherframe. Thus, for interpolation factors 0.2 and 0.4, the RGB pixel fromthe previous frame and the luminance pixel from the current frame areused for SAD calculations. For interpolation factors 0.6 and 0.8, RGBpixels from the current frame and luminance pixels from the previousframes are utilized. The worst case bandwidth required occurs at aninterpolation factor of 0.5.

FIG. 11 is a block diagram illustrating components of an example motionvector validation engine 250. In the configuration of FIG. 11, a centralprocessing unit 252 provides candidate motion vectors to a VMV buffer254. The VMV buffer 254 may be implemented as a circular buffer suchthat as one candidate motion vector is processed, a new motion vector isinput to the buffer 254. In the example of FIG. 11, the VMV buffer 254accommodates candidate motion vectors for 2 slices (up to 2×30 windows)so that the VMV engine 250 may work on one slice while the next slice isbeing written by the CPU 252. Using 18 bit motion vectors, the VMVbuffer 254 of this example is 17,280 bits in size.

The VMV buffer 254 provides motion vectors to a shift register 262. Theshift register 262 receives data associated with the previous framewindow through a double buffer 260. The previous frame window of FIG. 11is sized 96×62 pixels centered on the current 64×32 block to accommodatethe maximum motion vector excursions of (±32, ±16) pixels. The luminanceread block DMA 258 reads previous frame data from memory via the SDRAMcontroller 256. The double buffer configuration 260 enables the shifting262 and SAD processing 270 to be done at the same time as DMA operation258 for subsequent frame processing. The shift register 262 applies thecurrent candidate motion vector from the motion vector buffer 254 to theprevious frame data from the luminance window 260 to generate a shiftedcandidate window that is provided to the SAD calculator 270.

The SAD calculator 270 also receives current frame data via the currentframe read path 264, 266, 268. The RGB block read DMA 264 accessescurrent frame data via the SDRAM controller 256 and provides the currentframe data to the RGB window block 266. The RGB window block 266 ismultiple buffered similar to the luminance window 260 to facilitateparallel processing. In the example of FIG. 11, the RGB window buffer266 contains a window from the current frame with a size of 104×54pixels to accommodate motion vector excursions. The buffer 266 isimplemented as a multi-buffer so that SAD 270 and DMA 264 operations mayoccur simultaneously. The RGB block read DMA 264 is a high bandwidthsdram client that operates on a 256×54 pixel block for best efficiency.This requires that the RGB buffer 266 be capable of storing a total of552×54 pixels. At any point of time, VMV operations occur on a 104×54pixel section of this buffer. Data from the RGB buffer 266 may be passedthrough a low pass filter 268 prior to being provided to the SADcalculator 270.

The SAD calculator 270 receives the shifted candidate window from theshift register 262 and current frame data from the current frame readpath 264, 266, 268. The SAD calculator 270 performs a qualitycalculation on the shifted candidate frame and the current frame data bycomparing the two inputs. If the candidate motion vector associated withthe shifted candidate frame is accurate, the two inputs will be similarresulting in a low SAD score. If the candidate motion vector is poor,the two inputs will be very different resulting in a high SAD score. TheSAD calculator 270 selects the candidate motion vector associated withthe best quality calculation score as the final motion vector for apixel.

Following selection of a final motion vector, one or more intermediateframes are generated in the compensator block 272 and stored in thecompensation buffer 276, which may be implemented as a double buffer.The dynamic average block 278 receives the generated intermediate framesand applies error correction and concealment procedures. The dynamicaverage block 278 outputs the intermediate frames to a write-back buffer280 where the frames await being written to memory via the RGB blockwrite DMA 282.

FIG. 12 is a flow diagram illustrating a process 300 for calculatingmotion vectors from previous frame data 302 and current frame data 304inputs. At step 306, the received previous frame data 302 and currentframe data 304 are decomposed into high pass and low pass components.Fast Fourier transforms are then performed on each of the decomposedcomponents in step 308. Phase differences between the high passcomponents and between the low pass components are then calculated insteps 310 and 312, respectively. Following phase difference calculations310, 312, inverse fast Fourier transforms are performed on thecalculated phase differences in step 314. The inverse Fourier transformoutput is data that may be viewed as a phase plane correlation surface318. The output of the inverse fast Fourier transforms are utilized tocalculate candidate motion vectors in step 316. By identifying peaks onthe phase plane correlation surface 318 via a peak search 320, motionbetween the previous frame and the current frame can be detected. Basedon the location of these peaks, output motion vectors 322 are calculatedat 316.

The steps of FIG. 12 may be executed in a variety of orders with variedsteps including additional steps or the omission of certain steps. Forexample, steps 310 and 312 may be executed in parallel, or in series, ormay be computed at disparate times with other steps executed in between.Further, steps 314 and 316 may be executed for the high pass componentsfollowed by execution of steps 314 and 316 for the low pass components,or vice versa. Steps 314 and 316 could also be executed in parallel forthe low pass and high pass components as well.

FIG. 13 is a flow diagram illustrating a process 330 for generating anintermediate frame from previous frame data 332, current frame data 334,and additional motion vector inputs 342. At step 336, the receivedprevious frame data 332 and current frame data 334 are decomposed intohigh pass and low pass representations. These decomposed representationsare utilized to calculate a first motion vector estimate in step 338. Instep 340, a final motion vector is selected from the first motion vectoras well as one or more additional candidate vectors 342 based upon aquality calculation. Utilizing the selected final motion vector, anoutput intermediate frame 346 is generated based on the previous framedata 332 in step 344.

FIG. 14 is a block diagram illustrating a motion compensated picturerate conversion module 350 including a phase plane correlationcalculator 354 and a global motion calculator 356 that may be integratedas module 16 in the configurations of FIGS. 1 and 2, respectively, forproviding object motion estimation between pairs of consecutive framesin an input video signal 362.

In FIG. 14, an input video signal 362 is processed by a motionestimation engine 352 that includes the phase plane correlationcalculator 354 and the global motion calculator 356. The phase planecorrelation calculator 354 determines a motion vector estimate that isprovided to the SAD based vector validation module 358, as shown at 366.The motion vector estimate from the phase plane correlation calculator354 is also provided to the global motion calculator 356 to be used asan initial guess in calculating a second motion vector estimate that isprovided to the SAD based vector validation module 358, as shown at 368.The SAD based vector validation module 358 also may receive a copy ofthe video input signal 362 and a video information channel 364. The SADbased vector validation module 358 determines a per pixel motion vector374, 370 based upon the motion vector estimates 366, 368 from the phaseplane correlation calculator 354 and global motion calculator 356,respectively, and may consider other motion vector inputs (not shown) inthat determination. The per pixel motion vectors are received by aninterpolation and error concealment module 360, as shown at 370, whereintermediate frames are generated for insertion into a target video.

In estimating global motion, the most dominant motion in video istranslation. In an affine motion representation, the affine parametersthat represent the zoom and rotation are small in magnitude and may beestimated using a sparse motion field. Large translation estimation ismore difficult in comparison. Phase correlation based estimation, asdescribed above, measures sub-pixel motion with limited complexity.Using the predicted values from the phase correlation values as aninitial guess assuming no rotation or scaling in the image, thesubsequent RANSAC (RANdom SAmple Consensus) based estimation stagerefines the motion parameters.

The affine model is chosen for simplicity. However, the global motioncalculation may be extended to any other motion model such asprojective, etc. The affine motion model may be expressed as:X′=A*X+B,where X=[x,y]^(T) represents a position of a selected pixel group in theprevious frame, X′=[x′,y′]^(T) represents a position of a selected pixelgroup in the current frame, where

$A = \begin{bmatrix}a_{11} & a_{12} \\a_{21} & a_{22}\end{bmatrix}$represents motion parameters corresponding to zoom and rotation of theselected pixel group from the previous frame to the current frame, andB=[b _(x) ,b _(y)]^(T)represents motion parameters corresponding to pan motion of the pixelgroup from the previous frame to the current frame.

FIG. 15 is a flow diagram illustrating the calculation of affineparameters in a global motion calculator 380 that may be integrated asmodule 356 in the configuration of FIG. 14. The global motion calculator380 illustrates the two step process for generating a motion vectorestimate. The first step of the global motion calculation process 382produces a Kalman filter based prediction of the affine parameters byreceiving previous frame data and current frame data through a videoinput 386 and coarse motion vectors from the phase plane correlationcalculator, as shown at 388, to produce an initial guess 390, 392 at theaffine parameters. The second step 384 refines the initial guess 390,392 through a procedure such as a RANSAC based affine refinement toproduce refined affine parameters 394, 396.

FIG. 16 is a flow diagram illustrating a process for calculatingrefinement affine parameters. Following receipt of the current image 404and the previous image warped according to the phase plane correlationestimate, a number N_(i) of corresponding blocks from the warped image402 and current image 404 are selected in step 406. More specifically, aset of N_(i) random blocks is selected from the current image 404 thatalso have corresponding blocks in the warped previous image 402. Theselection of random blocks in step 406 tends to alleviate the problem ofsome new image regions appearing in an image frame and old regionsdisappearing from the image frame. The distribution of probability ofselecting a block from a region in an image may be configured to benon-uniform if some known segmentation information is available, so asto avoid selecting blocks from image regions which do not follow theglobal motion pattern. In practice, it is not necessary to pre-warp theentire previous image before starting the refinement stage, as it issufficient to warp only those parts of the previous image which areselected to participate in the refinement.

The selected blocks are sorted based upon their level of activity instep 408. In general, activity may be measured as a sum of absolutedifference from the mean, variance, or eigenvalues of a windowed imagesecond moment matrix. These three metrics are listed in increasing orderof computational complexity but also in increasing order of reliability.Another possibility is to look for a significant bi-direction intensitygradient inside a block. The sorting is intended to identify promisingblocks for motion estimation because low activity blocks are likely togive incorrect motion vectors due to aperture affect. The top N_(f)blocks from the sorted list are considered promising blocks and are keptunder consideration in step 410 while the remaining blocks arediscarded. Thus, an adaptive threshold on the activity is implicitlyapplied.

The translation between each of the high activity Nf promising blocks inthe current frame and its counterpart in the previous frame is measuredin step 412 based upon the phase correlation values calculated before.Phase-correlation provides two advantages as compared to other methods,in this regard. First, phase correlation measures sub-pixel-accuratetranslation with reasonably small amount of computations. Second, thetranslation measurement result is almost independent of minority outlierpixels, which may be due to foreground pixels. When neither backgroundnor foreground is dominant, it gives two motion vectors, onecorresponding to each.

After translation measurement is completed in step 412, the data ispassed to a RANSAC-based robust least-square model for fitting in step414 as a set of N_(f) pentuples (x, y, 1, dx, dy), where x,y is thecoordinate of the center of the block, dx is the translation of theblock along the x-axis, and dy is the translation of the block along they axis. Step 414 is described in further detail with reference to FIG.17 below. Following fitting in step 414, the refinement affineparameters 416 are output to be utilized in generating a motion vectorestimate.

FIG. 17 is a flow chart illustrating a process 420 for calculatingrefinement affine parameters. The process of FIG. 17 begins with thereceipt of a set of N_(f) data-points 422 and a normalization procedure424. The iteration index and maximum support variables are reset to zeroand the best affine parameters array is reset in step 426. A loop beginsat step 428 with an inquiry to check that a maximum number of iterationshas not been met. If a maximum number of iterations have not beencompleted, a random triplet of data-points is selected in step 432.Three data-points are selected because three-point correspondencescompletely define the affine transformation as long as the three pointsare not collinear. Each time through the loop, the selected threedata-points uniquely determine a set of affine parameters. Thecalculation of the affine parameters in step 434 may be calculated usingthe following relationship:

${\left\lbrack {\left( {A - I_{2}} \right)❘\frac{B}{k}} \right\rbrack = {\left\lbrack {\begin{matrix}{dx}_{1} \\{dy}_{1}\end{matrix}❘{\begin{matrix}{dx}_{2} \\{dy}_{2}\end{matrix}❘\begin{matrix}{dx}_{3} \\{dy}_{3}\end{matrix}}} \right\rbrack\left\lbrack {\begin{matrix}x_{1} \\y_{1} \\k\end{matrix}❘{\begin{matrix}x_{2} \\y_{2} \\k\end{matrix}❘\begin{matrix}x_{3} \\y_{3} \\k\end{matrix}}} \right\rbrack}^{- 1}},$where I₂ is a 2×2 identity matrix.

The parameter values that are calculated in step 434 may not beconsistent with affine parameters calculated over previous iterations ofthe loop beginning at step 428. If a data-point is consistent with aparticular value of affine parameters, then it is said to be in thesupport set of the affine parameter. Support for the current affineparameters is calculated in step 436. The number of supportingdata-points, which exhibit a motion vector consistent with the currentlydetermined affine parameters, is computed by counting the number ofdata-points for which the norm (e.g., L₁ or L₂) of the following errormetric is below a suitable threshold:

$e = {\begin{bmatrix}{dx} \\{dy}\end{bmatrix} - {{\left\lbrack {\left( {A - I_{2}} \right)❘\frac{B}{k}} \right\rbrack\begin{bmatrix}x \\y \\k\end{bmatrix}}.}}$

The “best affine parameter” refers to that value of affine parametersencountered so far that had the largest support set. The size of thesupport set for the best affine parameter is stored in the max supportvariable. Thus, in step 438, an inquiry is made as to whether thesupport for the current affine parameters is greater than the valuestored in the max support variable. If the current support is greater,the max support value is changed to the current support value, and thebest affine parameters array is reset to the current affine parametersvalue in step 442. If the current support is not greater, then thecurrent max support and best affine parameters values are maintained. Ineither case, the iteration index is incremented and the loop isrestarted as long as the iteration index is less than the max iterationsvalue.

Once the iteration index is equal to the maximum iterations value,branch 448 is taken and data-points not supporting the best affineparameters are discarded in step 450. A least-square plane fitting isapplied to the retained data-points in step 452 to determine the refinedaffine parameters 454 which are utilized to generate an estimated motionvector.

FIG. 18 is a flow diagram illustrating a process 460 for generating anintermediate frame from previous frame data 462 and current frame data464. At step 466, previous frame data 462 and current frame data 464 arereceived and a phase plane correlation calculation is performed todetermine a first motion vector estimate. The first motion vectorestimate is then utilized to perform a global motion computation todetermine a second motion vector estimate in step 468. In step 470, amotion compensated interpolation is performed to assign a final motionvector through optimization of a cost function. This motion compensatedinterpolation step 470 may utilize at least one of the first motionvector estimate and the second motion vector estimate. The assignedfinal motion vector and previous frame data are then used in step 472 togenerate an intermediate frame 474.

FIG. 19 is an illustration of a motion hole. As described previously,the determined final motion vector may not be valid for all pixels ofthe generated intermediate frame. These anomalies tend to occur nearedges of objects moving in a video. FIG. 19 depicts a motion hole thatresults from the invalidity of a final motion vector. The original shapeis shown at 512, and the original shape missing a region due to a motionhole is depicted at 514. To prevent the motion hole from appearing inthe intermediate frame, error correction procedures may be performed.

FIG. 20 is a flow diagram 480 illustrating an error concealmenttechnique. Block 482 receives the validated motion vector 488 for apixel or group of pixels from the VMV engine as well as current framedata 486 and previous frame data 484. A determination is made in theblock 482 as to whether the vector is valid. If the vector is found tobe valid 490, then motion compensated interpolation is performed at 492.If the vector is found to be invalid 494, then edge mask generationerror compensation 496 may be performed. Two different types of errorcompensation may be performed. If the area under consideration isconsidered detailed, such as a part of an edge between two objects in avideo, then branch 498 is taken and an edge propagation technique 500 isperformed. If the area under consideration is considered non-detailed,such as an area away from a transition between objects in a video, thenbranch 502 is taken and a texture synthesis technique 504 is performed.

FIG. 21 further illustrates a motion hole requiring error compensation520. A block 522 highlights an area that requires an error compensationprocedure. The block 522 includes some detailed portions in the shapeedge as well as some non-detailed portions in the fill in regions insideand outside of the shape.

The detailed portion of the target area 522 has an edge propagationtechnique applied. An edge map of the data is analyzed and the directionof the edge is computed in the neighborhood region near the target area522. Often, an edge direction is consistent over small distances. Thus,the edge pixels are copied from the neighborhood region and filled inthe hole based on a sum of squared differences or other matchingcriteria. Because detailed regions tend to have more information in theneighboring regions, edge propagation may be performed before texturesyntheses. Thus, the entire edge 524 may be restored prior to addressingany regions that require texture filling 526.

FIG. 22 is an illustration 530 of texture synthesis during an errorconcealment operation. Typically, non-detailed regions containstextures. Texture synthesis may be accomplished using a variety oftechniques. In the technique of FIG. 22, a poll is taken of blockssurrounding a portion of the fill region 534 to generate a set ofcandidate blocks 538. Based on the location of edges and the content ofsurrounding blocks, a best match 540 is selected to be incorporated intothe target area. Other more advanced techniques may take into accountthe periodicity of the surrounding textures as well as other parametersof the surrounding region.

One such advanced technique is depicted in FIG. 23. FIG. 23 is anillustration 550 of texture synthesis based on the Markov model. Underthe Markov model, it is assumed that the texture to be filled in israndom. The values of a number of surrounding pixels 552 that are notpart of the motion hole are examined. The Markov model uses aprobabilistic process to generate a most probable value 554 for thepixel of concern based on the value of surrounding pixels. This mostprobable value is then applied, and the process may be repeated for theremainder of the non-detailed target area to be filled.

While insertion of a motion compensated intermediate frame into anup-converted target video often improves the target video by increasingsharpness and reducing blur and judder, sometimes it may be preferableto repeat the previous frame when up-converting rather than using themotion interpolated frame. FIG. 24 is a flow diagram 560 illustrating afinal image synthesis decision. In FIG. 24, a multiplexer 562 selectsbetween a motion compensated interpolated frame 564 and a repeat of theprevious frame 566 based on a variety of criteria 568. For example, acalculation of the number of motion holes 570 created in generating theintermediate frame may be considered. Additionally, a quality of errorcompensation may be computed and considered in the decision whether touse the generated intermediate frame. Further, the magnitude of thefinal motion vectors 572 utilized in generating the intermediate framemay also be considered. If the magnitude of the motion vectors is small,then the advantage of the motion compensation may be nullified by anyerrors in the final compensated image. Thus, the statisticaldistribution 572 of the magnitude of the motion vectors may beconsidered in the decision.

FIG. 25 is a flow diagram 580 illustrating a process for generating aninterpolated frame from previous frame data 582 and current frame data584 input and deciding whether to use the interpolated frame in a targetvideo. The example of FIG. 25 begins at 586 where previous frame data582 and current frame data 584 are received, and a phase planecorrelation calculation is performed to determine a first motion vectorestimate. A global motion computation is then performed at step 588utilizing the first motion vector to determine a second motion vectorestimate. At step 590, a motion compensated interpolation is performedto assign a final motion vector via optimization of a cost function 590.Using the final motion vector and the previous frame data 582, anintermediate frame is generated in step 592. Following generation of anintermediate frame in step 592, an edge error concealing procedure isapplied to the intermediate frame in step 594. The edge errorconcealment step may include an edge interpolation procedure 596 as wellas a texture synthesis procedure 598. After the error concealment iscomplete, a decision whether to insert the intermediate frame in thetarget video or to repeat the previous frame in the target video iscomputed at step 600.

This written description uses examples to disclose the invention,including the best mode, and also to enable a person skilled in the artto make and use the invention. The patentable scope of the invention mayinclude other examples that occur to those skilled in the art.

1. A system for identifying motion between a previous frame and acurrent frame, the system comprising: a fast Fourier transformcalculator configured to generate corresponding low pass frequencydomain representations of previous frame data and current frame data andcorresponding high pass frequency domain representations of the previousframe data and the current frame data; a phase difference calculatorconfigured to calculate a first phase difference between the previousframe data low pass frequency domain representation and the currentframe data low pass frequency domain representation, the phasedifference calculator further configured to calculate a second phasedifference between the previous frame data high pass frequency domainrepresentation and the current frame data high pass frequency domainrepresentation; an inverse Fourier transform calculator configured togenerate a first inverse Fourier result based on the first phasedifference and a second inverse Fourier result based on the second phasedifference; and a motion vector calculator configured to generate twomotion vectors based on the first inverse Fourier result and the secondinverse Fourier result; wherein the generated two motion vectors areused to generate a display on a display device.
 2. The system of claim1, further comprising a decomposition filter that receives the previousframe data and the current frame data, the decomposition filtercomprising a high pass filter and a low pass filter, the decompositionfilter configured to provide a high pass filter output and a low passfilter output to the fast Fourier transform calculator.
 3. The system ofclaim 1, wherein the first inverse Fourier result is a first phase planecorrelation surface, and wherein the second inverse Fourier result is asecond phase plane correlation surface.
 4. The system for claim 3,wherein the motion vector calculator comprises a peak searcher thatidentifies a first peak on the first phase plane correlation surface anda second peak on the second phase plane correlation surface.
 5. Thesystem of claim 4, wherein the two motion vectors correspond to alocation of the first identified peak and a location of the secondidentified peak.
 6. The system of claim 5, wherein the locations of thefirst identified peak and the second identified peak correspond to amovement in a video from the previous frame to the current frame.
 7. Thesystem of claim 4, wherein the peak searcher identifies a firstplurality of additional peaks on the first phase plane correlationsurface and a second plurality of additional peaks on the second phaseplane correlation surface.
 8. The system of claim 2, wherein theprevious frame data and the current frame data correspond toframe-blocks 64×32 in size, and wherein the low pass filter is amulti-tap filter having a block size of 15×7.
 9. The system of claim 2,wherein the high pass filter comprises a gradient edge based detector.10. The system of claim 9, wherein the gradient edge based detector is aSobel edge detector.
 11. The system of claim 1, wherein the fast Fouriertransform calculator utilizes a radix 2 decimation-in-timeimplementation.
 12. The system of claim 1, wherein the fast Fouriertransform calculator has an input resolution of 2×16 bits and an outputresolution of 2×16 bits.
 13. The system of claim 1, wherein the phasedifference calculator uses a CORDIC implementation, wherein the firstphase difference and the second phase difference are represented with 8bit resolution, and the phase difference calculator has a latency of 6clock cycles.
 14. The system of claim 1, wherein the phase differencecalculator comprises a phase subtractor and a lookup table.
 15. A methodof identifying motion between a previous frame and a current frame, themethod comprising: decomposing received previous frame data and currentframe data into a high pass component and a low pass component;performing a fast Fourier transform of the high pass components and lowpass components of the previous frame data and the current frame data;calculating a first phase difference between the high pass component ofthe current frame data and the high pass component of the previous framedata; calculating a second phase difference between the low passcomponent of the current frame data and the low pass component of theprevious frame data; performing an inverse fast Fourier transform on thefirst phase difference and the second phase difference to generate afirst inverse Fourier result and a second inverse Fourier result; andcalculating a first motion vector and a second motion vector based onthe first inverse Fourier result and the second inverse Fourier result;wherein the generated two motion vectors are used to generate a displayon a display device.
 16. The method of claim 15, wherein the firstinverse Fourier result is a first phase plane correlation surface, andthe second inverse Fourier result is a second phase plane correlationsurface.
 17. The method of claim 16, wherein calculating a first motionvector and a second motion vector further comprises searching for afirst peak of the first phase plane correlation surface and a secondpeak of the second phase plane correlation surface.
 18. The method ofclaim 17, wherein the first motion vector and the second motion vectorare calculated based on locations of the first peak on the first phaseplane correlation surface and the second peak of the second phase planecorrelation surface.
 19. The method of claim 18, wherein searching forpeaks comprises identifying a maximum value in a 3×3 neighborhood, andzeroing all non-maximum values in the 3×3 neighborhood.
 20. The methodof claim 18, wherein searching for peaks further comprises identifying afirst plurality of peaks from the first phase plane correlation surfaceand a second plurality of peaks from the second phase plane correlationsurface.